System and method for unsupervised and active learning for automatic speech recognition

ABSTRACT

A system and method is provided for combining active and unsupervised learning for automatic speech recognition. This process enables a reduction in the amount of human supervision required for training acoustic and language models and an increase in the performance given the transcribed and un-transcribed data.

PRIORITY

The present application is a continuation of U.S. patent applicationSer. No. 12/414,587, filed Mar. 30, 2009, which is a continuation ofU.S. patent application Ser. No. 10/742,854, filed Dec. 23, 2003, nowU.S. Pat. No. 7,533,019, the contents of which is incorporated herein byreference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to speech recognition systemsand, more particularly, to an unsupervised and active learning processfor automatic speech recognition systems.

2. Introduction

State-of-the-art speech recognition systems require transcribedutterances for training of various knowledge sources, such as languageand acoustic models, used in the speech recognition system. In general,acoustic models can include the representation of knowledge aboutacoustics, phonetics, microphone and environment variability, gender anddialect differences among speakers, etc. Language models, on the otherhand, can refer to a system's knowledge of what constitutes a possibleword, what words are likely to co-occur, and in what sequence. Thesemantics and functions related to an operation a user may wish toperform may also be necessary for the language model.

Many uncertainties exist in automatic speech recognition (ASR). Forexample, uncertainties may relate to speaker characteristics, speechstyle and rate, recognition of basic speech segments, possible words,likely words, unknown words, grammatical variation, noise interference,and normative accents. Each of these uncertainties can be the cause inthe reduction of recognition success in an ASR system. A successfulspeech recognition system must therefore contend with all of theseissues.

A speech recognition system generally seeks to minimize uncertaintiesthrough the effective training of the system. As noted, this training isbased on the generation of transcribed utterances, a process that islabor intensive and time-consuming process. As would be appreciated, ifissues of cost were ignored, more effective speech recognition systemscould be created through the use of greater amounts of transcribed data.This does not represent a practical solution. What is needed thereforeis an efficient mechanism for creating a quality speech recognitionsystem using all available sources of training data, whether existing intranscribed or un-transcribed form.

SUMMARY

In accordance with the present invention, a process is provided forcombining active and unsupervised learning for automatic speechrecognition. This process enables a reduction in the amount of humansupervision required for training acoustic and language models and anincrease in the performance given the transcribed and un-transcribeddata.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates an exemplary method of the present invention;

FIG. 2 illustrates a scatter plot of n-gram probabilities estimated fromclean vs. noisy speech utterance transcriptions;

FIG. 3 illustrates the false rejection rate versus false acceptancerate;

FIG. 4 illustrates the percentage of correctly recognized words inconfidence score bins;

FIG. 5 illustrates the percentage of correctly recognized words inconfidence score bins;

FIG. 6 illustrates the vocabulary size growth with random and selectivesampling; and

FIG. 7 illustrates word accuracy learning curves.

DETAILED DESCRIPTION

Various embodiments of the invention are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the invention.

As noted, state-of-the-art automatic speech recognition (ASR) systemsrequire transcribed utterances for training Utterances may be obtainedfrom recorded conversations between customers and a call center. Adeveloper or a team of developers must transcribe and label theutterances to identify user intent. Not every utterance is valuable,however. For example, if a user calls a call center to order atelevision, he may talk with the call center employee about the weatherbefore saying “I'd like to order a TV.”

The transcription process used to generate these transcribed utterancesis labor intensive and time-consuming. One method of reducing the numberof training examples to be labeled is through active learning, whichinspects the unlabeled examples, and selectively samples the mostinformative ones with respect to a given cost function for a human tolabel. One of the goals of the active learning algorithm is to selectthe examples for labeling that will have the largest performanceimprovement. Another method of reducing the cost of training an ASRsystem is through unsupervised learning, which aims at utilizingunlabeled examples, to train, augment, or adapt the system.

In accordance with the present invention, the transcription effort fortraining an ASR system is reduced through the combination of active andunsupervised learning techniques. As will be described, this processuses word and utterance confidence scores for utterance selection inactive learning, and probability estimation in unsupervised learning. Inthe description following below, the main active and unsupervisedlearning processes are briefly described first, followed by adescription of their combination.

Active learning is generally designed to aid and automate thelabor-intensive process of building and training speech recognitionmodels. One of the goals of an active learning system is tosignificantly reduce the amount of transcribed data required to trainASR models with a given level of accuracy. This reduction in transcribeddata will ultimately reduce the cost and time-to-market for naturallanguage services.

In the active learning process, a speech recognizer is first trainedusing a small set of transcribed data S_(t). This small set may bemanually generated from a group of utterances or automatically generatedfrom web pages related to the application domain. Using the recognizer,the system recognizes the utterances that are candidates fortranscription Su, where S_(u) is an additional un-transcribed set. Usinglattice-based confidence measures, the system predicts which candidatesare recognized incorrectly. The human transcriber then transcribes theutterances that are most likely to have recognition errors.

In unsupervised learning, one of the goals is exploiting un-transcribeddata to either bootstrap a language model or in general improve upon themodel trained from the transcribed set of training examples. In otherwords, the model trained from the transcribed set of training examplesis augmented using machine-labeled data rather than human-labeled data.

FIG. 1 illustrates an embodiment of a combined unsupervised and activelearning process for automatic speech recognition. As illustrated, theprocess begins at step 102 through the training of the initial acousticand language models AM′ and LM′ for recognition using a small set oftranscribed data S_(t) and also possibly a set of availableun-transcribed utterances. Here, the variable “i” is the iterationnumber, and as shown in step 102, i=0 for the initially trained acousticand language models AM′ and LM′. Next, at step 104, the utterances inset S_(u) are recognized using the trained acoustic and language models.At step 106, the confidence scores are computed for all of the words orun-transcribed utterances. Before continuing with the flow chart of FIG.1, the calculation of confidence scores will be discussed.

Confidence scores can be calculated in various ways. In a firstembodiment, confidence scores are calculated based on acousticmeasurements. In a second embodiment, confidence scores are calculatedbased on word lattices. The second embodiment has the advantage that theprobability computation does not require training of an estimator. Aswould be appreciated, hybrid approaches that use features of these twoembodiments can also be used.

In accordance with the present invention, word confidence scores areextracted from the lattice output of the ASR system. In general, theword confidence score extraction process comprises: computing theposterior probabilities for all transitions in the lattice; extracting apath from the lattice (which can be the best, longest or a random path,which path is called the pivot of the alignment); traversing thelattice; aligning all the transitions with the pivot; and merging thetransitions that correspond to the same word (or label) that occur inthe same interval (by summing their posterior probabilities). A detailedexplanation of this process and the comparison of its performance withother approaches is presented in Hakkani-Tur et al., A General Algorithmfor Word Graph Matrix Decomposition, Proceedings of ICASSP, Hong Kong,April 2003, which is incorporated by reference herein in its entirety.

The state times or approximate state locations on the lattice are usedto align transitions that have occurred at around the same timeinterval. The final structure is called a pivot alignment. The wordposterior probability estimates on the best path of the pivot alignmentsare used as word confidence scores, c,.

For active learning, different approaches were evaluated to obtainutterance level confidence measures from word confidence scores that areextracted from ASR lattices. One approach is to compute the confidencescore of an utterance as the arithmetic mean of the confidence scores ofthe words that it contains:I^(n)

$\begin{matrix}{{{{c\text{,,}} = {- {lc}}};}n} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

Another approach is to compute the confidence score of an utterance asthe product of the confidence scores of the words that it contains:

$\begin{matrix}{{{c\text{,,}} = {Ica}^{\prime(^{\prime\prime\prime}\prime)}}{1 = 1}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$where a,(14),) is a scaling function. Other functions can also be usedto compute the utterance confidence scores:c _(w,),,=F(c,, . . . , c _(n))  (Equation 3)where F can be the geometric mean or the min function.

Returning to the flowchart of FIG. 1, after the confidence scores arecomputed at step 106, k utterances are selected, at step 108, that havethe smallest confidence scores from S_(u). Next, at step 110, the kselected utterances are transcribed by human effort. At step 112, thenew k selected utterances that are transcribed and denoted by Sk, areadded to the transcribed set S_(t) (i.e., S,I+1=^(s:)U Ski), while alsobeing subtracted from the set of available utterances S_(u) (i.e.,S′7¹=S_(u) ¹−S_(k) ^(i)). The new set of training data S;+₁ can then beused to train the new acoustic and language models AM′^(±1) and LIV1¹⁺¹at step 114.

As illustrated, the additional un-transcribed set S_(u) is madeavailable to the ASR system processing at steps 104 and 112. The initialtranscribed set S_(t) is also made available to the initial trainingstep 102, recalculation step 112, and training step 114.

At step 116, a determination is also made as to whether the wordaccuracy (WER=word error rate) has converged. If it is determined thatthe word accuracy has not converged, then the process loops back to step104. If it is determined that the word accuracy has converged, then theprocess ends as further transcription is not expected to improve ASRperformance. The process can be repeated again later, as newun-transcribed data from the domain becomes available.

As further illustrated in FIG. 1, un-transcribed data S_(u) can also beused to train the acoustic and language models AM^(I+1) and LM′^(±1) atstep 114. More specifically, the ASR output and word confidence scoresfor un-transcribed data are used to train, augment, or adapt the ASRsystem. As illustrated in FIG. 1, in one embodiment, only a selectedsample (chosen at step 118) of un-transcribed data are used to train,augment, or adapt the ASR system. This element of unsupervised learningis therefore used in combination with the active learning processdescribed above.

One issue with unsupervised learning is the estimation of the errorsignal. In the case of language modeling the error signal is the noiseon the event counts. Even in the simple case of n-gram language modelingthe n-gram counts in the presence of noise are very unstable. FIG. 2illustrates a scatter a plot of n-gram probabilities as estimated fromclean (true) transcriptions of speech (x axis) and estimated from noisytranscriptions (y axis), in this case the ASR output. As illustrated,FIG. 2 shows how the error variance increases for infrequent n-grams.

In standard n-gram estimation the occurrences of n-tuples C(w_(i) ^(n))are counted, where w_(i) ^(n) is the word n-tuple wi, _(W23 . . . /wn).In unsupervised learning the nature of the information is noisy andn-gram counts are estimated from two synchronized information channels,the speech utterance hypothesis and the synchronized error signal. Foreach word w,, the probability of being correctly decoded c, (=1−e,,where e, is the error probability) is estimated, resulting in theconfidence score. The bidimensional channel is then represented as asequence of n-tuples of symbol pairs (w_(i) ^(n),c_(i)^(n))=(wi,c1)(w2,c2), . . . , (wn,cn). The n-gram counts in presence ofnoise can be computed by marginalizing the joint channel counts:

$\begin{matrix}{{{{{CuL}\left( {{w;}\;}^{\prime} \right)} = {c_{x}6_{,\; 4}}},(x)}{xET}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$where c,, is the confidence score for the n-tuple x in the noisy spokenutterance transcriptions T and 6,, (x) is the indicator function for then-tuple w_(i) ^(n). The confidence score of the n-tuple w_(i) ^(n) canbe computed by geometric or arithmetic means or max and min over then-tuple of word confidence scores c,”. In one embodiment, the simplestapproach is taken where c_(v4),=c₁₂ is computed. Equation 4 can berewritten as a function of the error probability e_(n):

$\begin{matrix}{{{{\,^{C}{UL}}\left( {{}_{}^{}{}_{}^{}} \right)} = {{\,^{C}\left( {}^{W}i^{n} \right)} - {0_{4}(x)}}}{xET}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

Equation 5 shows the relation between the count estimates with andwithout error signal Cu_(L)(w,″) and C(w_(i) ^(n)), respectively.

A series of experiments verified that word confidence scores can be usedto identify correctly recognized words, utterance confidence scores canbe used to select more informative utterances to transcribe, andautomatic speech recognition accuracy can be improved by exploitingun-transcribed data. For all these experiments, utterances from AT&T'sHow May I Help You?^(sm) (HMIHY) speech database was used. See A. Gorinet al., “Semantic Information Processing of Spoken Language, In Proc. ofATR Workshop on Multi-Lingual Speech Communication. 2000, which isincorporated herein by reference in its entirety. The language modelsused in all experiments were trigram models based on Variable NgramStochastic Automata. See G. Riccardi et al., “Stochastic Automata forLanguage Modeling,” Computer Speech and Language, 10:265-293, 1996,which is incorporated herein by reference in its entirety.

In the HMIHY speech database there were two distinct data collections.The first is from human-human conversations (8K utterances and 300K wordtokens) and consists of responses to the initial prompt. The second isfrom human-machine dialogs (28K and 318K word tokens) and consists ofusers' responses to all system prompts (e.g. greeting and confirmationprompts). The test data consists of 1,000 utterances (10K words) fromthe human-machine data collection. All experiments kept the acousticmodel fixed, and used a model, which is subword unit based, withtriphone context modeling, and trained using utterances from human-humanconversations, and other telephone speech. The training data for theacoustic model did not overlap with the additional training data.

The language model that was used was trained using all the human-humanutterances to recognize the test data, and computed word confidencescores for the ASR output. The word accuracy of the test set was 63.8%.To check how good the word confidence scores were in distinguishing thecorrectly recognized and misrecognized words, a binary classificationproblem was considered, where the confidence scores of the words wasused, as well as a threshold for the decision. Each word was classifiedas correctly recognized if that word had a confidence score higher thanthe threshold, and as misrecognized otherwise. False rejection and falseacceptance rates were computed by changing the threshold and comparingthe classifications with the references (see FIG. 3). False rejectionrate is the percentage of correctly recognized words that are classifiedas misrecognition, since their confidence score is lower than thethreshold. False acceptance rate is the percentage of misrecognizedwords that are accepted as correct recognitions, since their confidencescore is higher than the threshold. On the test set, the equal errorrate (False Rejection Rate equals False Acceptance Rate) is 22%.

Alternatively, FIG. 4 also plots the percentage of correctly recognizedwords in confidence score bins. As illustrated, for example, 8.1% of thewords that have a confidence score in the interval [0.1,0.2] are correctrecognitions, and the rest are misrecognitions. One may expect k % ofthe words having the confidence score of k/100 to be correct. As seen,the percentage of correctly recognized words in each confidence scorebin increases almost linearly as the confidence score increases.

Utterance confidence scores are computed from word confidence scores forselectively sampling the utterances. FIG. 5 shows a scatter plot ofutterance confidence score versus the percentage of correctly recognizedwords in that utterance. As illustrated, as the confidence score of theutterances increase, the percentage of correctly recognized words alsoincrease, showing that good quality utterance confidence scores can becomputed using word confidence scores.

To show the effectiveness of the estimates of the noisy counts n-gramlanguage models were trained to run large vocabulary speech recognitionexperiments. A baseline stochastic language model (SLM) was trained from1K clean human-human speech transcriptions (baseline) and then augmentedwith 20K noisy transcriptions with and without error signal (ASR outputand Confidence Scores). Table 1 shows the Word Accuracy (WA) results on1K test set taken from the HMIHY database. Using the counts estimatedusing equation 5 a 30% reduction of the gap between the baseline andsupervised SLM (Upper bound) was achieved without requiring furthertranscription of speech utterances.

TABLE 1 Training Set WA Baseline 59.1% ASR output + baseline 61.5% ASRoutput + Confidence Scores + baseline 62.1% Upper bound 69.4%

For active learning in ASR, an initial language model was trained usingthe initial set utterances from human-human interactions. Using thismodel, lattices and pivot alignments were then generated for theadditional training data, and the confidence scores for words andutterances were then computed. Using the confidence scores forutterances, the utterances were then sorted in an order according to“usefulness” for ASR (generating the selectively sampled order). Thelanguage models were incrementally trained by using the top n utterancesfrom the randomly sampled and selectively sampled orders, and learningcurves were generated for word accuracy and vocabulary size (See FIGS. 6and 7). The experiments used the arithmetic mean of the word confidencescores (i.e., F is the mean function in equation 3) as utteranceconfidence scores, which gave the best results. The active learningprocess is independent of the way the confidence scores are computed. Inthe experiments, the normalized utterance likelihood was also used as asampling criterion, and it gave inferior performance.

FIG. 7 also shows a learning curve when active and unsupervised learningare combined. For this curve, transcriptions of the top n utteranceswere added to the selectively sampled order, and the automatic speechrecognizer output of the rest of the utterances to the training set forthe language models. The recognizer output was generated using theinitial language model, and contained word confidence scores. FIG. 7shows the word accuracy learning curves when additional un-transcribeddata was exploited by combining active learning with unsupervisedlearning.

These curves illustrate that active learning is effective in reducingthe need for labeled data (for a given word accuracy). For example, toachieve 66.5% word accuracy with random sampling, transcriptions for4,000 utterances were needed, however, this accuracy can be achieved bytranscribing only around 2,500 utterances. This shows that the sameperformance is achieved by transcribing 38% fewer utterances, whenactive learning is used. In addition, a huge improvement is obtainedusing the un-transcribed data, at the very initial points. For example,the combination of active and unsupervised learning produces a 66.5%word accuracy by using only 1,000 transcribed utterances instead of4,000 transcribed utterances, that is 75% less utterances than randomsampling. As more transcribed data is produced, the improvement usingun-transcribed data gets less and active learning takes over. Thecombination always results in higher word accuracy than random sampling,by 2-3% points.

As thus described, new methods are provided for reducing the amount oflabeled training examples by selectively sampling the most informativesubset of data for transcription using lattice based confidencemeasures, and exploiting the rest of the data, that has not beentranscribed, by using the ASR output and word confidence scores. Byselective sampling using utterance-level confidence measures, the sameword accuracy results are achieved using 38% less data. It has beenshown that it is possible to detect utterances that have little newinformation when added to an initial set of utterances. In addition tothis, it has been shown that it is possible to exploit theun-transcribed data, and the same word accuracy results can be achievedusing 75% less data by combining active and unsupervised learning.

Using the principles of the present invention, an improved process isprovided for creating an ASR module. Thus, the present inventionenvisions a further embodiment that includes a spoken dialog servicehaving an ASR module generated by the principles of the presentinvention.

Embodiments within the scope of the present invention may also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer. By way of example, and notlimitation, such computer-readable media can comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to carryor store desired program code means in the form of computer-executableinstructions or data structures.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, components, and data structures, etc. that perform particulartasks or implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of theinvention may be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote memory storage devices.

Although the above description may contain specific details, they shouldnot be construed as limiting the claims in any way. Other configurationsof the described embodiments of the invention are part of the scope ofthis invention. For example, the preferred embodiments of the inventionmay be described with reference to ASR components within a spoken dialogsystem. However, the invention may have applicability in a variety ofenvironments where ASR may be used. Therefore, the invention is notlimited to ASR within any particular application. Accordingly, theappended claims and their legal equivalents only should define theinvention, rather than any specific examples given.

The invention claimed is:
 1. A method comprising: identifying, in adatabase of utterances, first transcribed utterances and firstun-transcribed utterances; identifying transcription candidateutterances from the first un-transcribed utterances; predicting, via aprocessor, likely recognition error utterances from the transcriptioncandidate utterances; forwarding the likely recognition error utterancesto a transcriber, to yield additional transcribed utterances; adding theadditional transcribed utterances to the database of utterances, toyield an updated database of second transcribed utterances and secondun-transcribed utterances; and if word accuracy has converged, modifyingan automatic speech recognition system by providing both the secondtranscribed utterances and the second un-transcribed utterances to theautomatic speech recognition system.
 2. The method of claim 1, whereinidentifying transcription candidate utterances further comprises usingconfidence scores of the un-transcribed utterances.
 3. The method ofclaim 2, the method further comprising: determining the confidencescores using an acoustic model and a language model.
 4. The method ofclaim 1, wherein word posterior probability estimates are used forconfidence scores associated with the database of utterances.
 5. Themethod of claim 1, wherein a word is considered to be correctlyrecognized if the word has a confidence score higher than a thresholdvalue.
 6. The method of claim 5, wherein the confidence score is basedon at least one of a scaling function and a geometric mean.
 7. Themethod of claim 1, further comprising: upon adding the additionaltranscribed utterances to the database of utterances, removing theadditional transcribed utterances from the first un-transcribedutterances to yield the second un-transcribed utterances.
 8. A systemcomprising: a processor; a memory storing instructions for controllingthe processor to perform steps comprising: identifying, in a database ofutterances, first transcribed utterances and first un-transcribedutterances; identifying transcription candidate utterances from thefirst un-transcribed utterances; predicting likely recognition errorutterances from the transcription candidate utterances; forwarding thelikely recognition error utterances to a transcriber, to yieldadditional transcribed utterances; adding the additional transcribedutterances to the database of utterances, to yield an updated databaseof second transcribed utterances and second un-transcribed utterances;and if word accuracy has converged, modifying an automatic speechrecognition system by providing both the second transcribed utterancesand the second un-transcribed utterances to the automatic speechrecognition system.
 9. The system of claim 8, wherein identifyingtranscription candidate utterances further comprises using confidencescores of the un-transcribed utterances.
 10. The system of claim 9, thesteps further comprising: determining the confidence scores using anacoustic model and a language model.
 11. The system of claim 8, whereinword posterior probability estimates are used for confidence scoresassociated with the database of utterances.
 12. The system of claim 8,wherein a word is considered to be correctly recognized if the word hasa confidence score higher than a threshold value.
 13. The system ofclaim 12, wherein the confidence score is based on at least one of ascaling function and a geometric mean.
 14. The system of claim 8, thesteps further comprising: upon adding the additional transcribedutterances to the database of utterances, removing the additionaltranscribed utterances from the first un-transcribed utterances to yieldthe second un-transcribed utterances.
 15. A non-transitorycomputer-readable storage medium storing instructions which, whenexecuted by a computing device, cause the computing device to performsteps comprising: identifying, in a database of utterances, firsttranscribed utterances and first un-transcribed utterances; identifyingtranscription candidate utterances from the first un-transcribedutterances; predicting likely recognition error utterances from thetranscription candidate utterances; forwarding the likely recognitionerror utterances to a transcriber, to yield additional transcribedutterances; adding the additional transcribed utterances to the databaseof utterances, to yield an updated database of second transcribedutterances and second un-transcribed utterances; and if word accuracyhas converged, modifying an automatic speech recognition system byproviding both the second transcribed utterances and the secondun-transcribed utterances to the automatic speech recognition system.16. The non-transitory computer-readable storage medium of claim 15,wherein identifying transcription candidate utterances further comprisesusing confidence scores of the un-transcribed utterances.
 17. Thenon-transitory computer-readable storage medium of claim 16, the stepsfurther comprising: determining the confidence scores using an acousticmodel and a language model.
 18. The non-transitory computer-readablestorage medium of claim 15, wherein word posterior probability estimatesare used for confidence scores associated with the database ofutterances.
 19. The non-transitory computer-readable storage medium ofclaim 15, wherein a word is considered to be correctly recognized if theword has a confidence score higher than a threshold value.
 20. Thenon-transitory computer-readable storage medium of claim 19, wherein theconfidence score is based on at least one of a scaling function and ageometric mean.